<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" href="../../style.css" type="text/css" media="screen">
<link rel="stylesheet" href="../../print.css" type="text/css" media="print">
<meta content="Tokenizer,memorphic.parser.Tokenizer,token,rawData,addTokenPattern,buildRegExpORList,buildRegExpORList2,createToken,getState,getTokenMetrics,lookAhead,nextToken,parseNextToken,reset,restoreState,skipAhead,skipToken,tokenWasSkipped" name="keywords">
<title>memorphic.parser.Tokenizer</title>
</head>
<body>
<script type="text/javascript" language="javascript" src="../../asdoc.js"></script><script type="text/javascript" language="javascript" src="../../cookies.js"></script><script type="text/javascript" language="javascript">
<!--
				asdocTitle = 'Tokenizer - API Documentation';
				var baseRef = '../../';
				window.onload = configPage;
			--></script>
<table style="display:none" id="titleTable" cellspacing="0" cellpadding="0" class="titleTable">
<tr>
<td align="left" class="titleTableTitle">API Documentation</td><td align="right" class="titleTableTopNav"><a onclick="loadClassListFrame('../../all-classes.html')" href="../../package-summary.html">All&nbsp;Packages</a>&nbsp;|&nbsp;<a onclick="loadClassListFrame('../../all-classes.html')" href="../../class-summary.html">All&nbsp;Classes</a>&nbsp;|&nbsp;<a onclick="loadClassListFrame('../../index-list.html')" href="../../all-index-A.html">Index</a>&nbsp;|&nbsp;<a href="../../index.html?memorphic/parser/Tokenizer.html&amp;memorphic/parser/class-list.html" id="framesLink1">Frames</a><a onclick="parent.location=document.location" href="" style="display:none" id="noFramesLink1">No&nbsp;Frames</a></td><td rowspan="3" align="right" class="titleTableLogo"><img alt="Adobe Logo" title="Adobe Logo" class="logoImage" src="../../images/logo.jpg"></td>
</tr>
<tr class="titleTableRow2">
<td align="left" id="subTitle" class="titleTableSubTitle">Class&nbsp;Tokenizer</td><td align="right" id="subNav" class="titleTableSubNav"><a href="#propertySummary">Properties</a>&nbsp;| <a href="#methodSummary">Methods</a></td>
</tr>
<tr class="titleTableRow3">
<td colspan="2">&nbsp;</td>
</tr>
</table>
<script type="text/javascript" language="javascript">
<!--
if (!isEclipse() || window.name != ECLIPSE_FRAME_NAME) {titleBar_setSubTitle("Class Tokenizer"); titleBar_setSubNav(false,true,false,false,false,false,true,false,false,false,false,false,false,false);}
--></script>
<div class="MainContent">
<table cellspacing="0" cellpadding="0" class="classHeaderTable">
<tr>
<td class="classHeaderTableLabel">Package</td><td><a onclick="javascript:loadClassListFrame('class-list.html')" href="package-detail.html">memorphic.parser</a></td>
</tr>
<tr>
<td class="classHeaderTableLabel">Class</td><td class="classSignature">public class Tokenizer</td>
</tr>
<tr>
<td class="classHeaderTableLabel">Subclasses</td><td><a href="../xpath/parser/XPathTokenizer.html">XPathTokenizer</a></td>
</tr>
</table>
<p></p>
  
  Outstanding issues:
  
   1. Lookahead is not possible in determining if a token should be discarded. Currently this 
    causes tokenPosition to become out of sync, because the lookAhead token is given the 
    index tokenPosition+1, but the removal of the previous token means that this index
    is 1 too high. My initial solution was to produce flagged "skip" tokens, which would 
    be ignored by nextToken(), which would cause it to invoke itself again, but that caused
    some parse errors, that I could not identify the cause of.
  
   2. There is no enforcment that the input stream is completely tokenized and that those 
    tokens are all consumed. The solution is probably to return an EOFToken when the 
    inputStream is exhausted and the AST grammar can check for the existance of that token 
    after the last pattern is matched. Currently SyntaxTree.verifyTree() is checking for
    bytesRemaining()>0, which is probably not reliable. SyntaxTree could also test for 
    an EOFError being thrown.
  
  <p></p>
<br>
<hr>
</div>
<a name="propertySummary"></a>
<div class="summarySection">
<div class="summaryTableTitle">Public Properties</div>
<table id="summaryTableProperty" class="summaryTable " cellpadding="3" cellspacing="0">
<tr>
<th>&nbsp;</th><th colspan="2">Property</th><th class="summaryTableOwnerCol">Defined&nbsp;by</th>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol"><a class="signatureLink" href="#rawData">rawData</a> : String<div class="summaryTableDescription">
   
   
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
</table>
</div>
<a name="protectedPropertySummary"></a>
<div class="summarySection">
<div class="summaryTableTitle">Protected Properties</div>
<table id="summaryTableProtectedProperty" class="summaryTable " cellpadding="3" cellspacing="0">
<tr>
<th>&nbsp;</th><th colspan="2">Property</th><th class="summaryTableOwnerCol">Defined&nbsp;by</th>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol"><a class="signatureLink" href="#token">token</a> : <a href="../parser/Token.html">Token</a>
<div class="summaryTableDescription">
   The token that was most recently created
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
</table>
</div>
<a name="methodSummary"></a>
<div class="summarySection">
<div class="summaryTableTitle">Public Methods</div>
<table id="summaryTableMethod" class="summaryTable " cellpadding="3" cellspacing="0">
<tr>
<th>&nbsp;</th><th colspan="2">Method</th><th class="summaryTableOwnerCol">Defined&nbsp;by</th>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#Tokenizer()">Tokenizer</a>()</div>
<div class="summaryTableDescription"></div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#addTokenPattern()">addTokenPattern</a>(pattern:<a href="TokenPattern.html">TokenPattern</a>):void</div>
<div class="summaryTableDescription">
   Adds the definition of a new TokenPattern, used for parsing the input string into tokens
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#getState()">getState</a>():<a href="../parser/TokenizerState.html">TokenizerState</a>
</div>
<div class="summaryTableDescription">
   Captures the state of  the tokenizer so that it may be restored later.</div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#getTokenMetrics()">getTokenMetrics</a>(token:<a href="Token.html">Token</a>):<a href="../parser/TokenMetrics.html">TokenMetrics</a>
</div>
<div class="summaryTableDescription">
   This is not an optimized way of getting this information; it's really just for debugging
   and error messages
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#lookAhead()">lookAhead</a>(offset:int = 1):<a href="../parser/Token.html">Token</a>
</div>
<div class="summaryTableDescription">
   Get a subsequent token, without affecting the tokenizer state
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#nextToken()">nextToken</a>():<a href="../parser/Token.html">Token</a>
</div>
<div class="summaryTableDescription">
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#reset()">reset</a>():void</div>
<div class="summaryTableDescription">
   Start getting tokens from the beginning
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#restoreState()">restoreState</a>(state:<a href="TokenizerState.html">TokenizerState</a>):void</div>
<div class="summaryTableDescription">
   restores the tokenizer state to a memento(TM) created from getState()
   </div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#skipAhead()">skipAhead</a>(numTokens:int = 1):void</div>
<div class="summaryTableDescription">
   Skip the specified number of tokens.</div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
</table>
</div>
<a name="protectedMethodSummary"></a>
<div class="summarySection">
<div class="summaryTableTitle">Protected Methods</div>
<table id="summaryTableProtectedMethod" class="summaryTable " cellpadding="3" cellspacing="0">
<tr>
<th>&nbsp;</th><th colspan="2">Method</th><th class="summaryTableOwnerCol">Defined&nbsp;by</th>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#buildRegExpORList()">buildRegExpORList</a>(str:String):String</div>
<div class="summaryTableDescription">[static]</div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#buildRegExpORList2()">buildRegExpORList2</a>(... rest):String</div>
<div class="summaryTableDescription">[static]</div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#createToken()">createToken</a>(pattern:<a href="TokenPattern.html">TokenPattern</a>, value:String, sourceIndex:int, prev:<a href="Token.html">Token</a> = null):<a href="../parser/Token.html">Token</a>
</div>
<div class="summaryTableDescription">
   Factory method for Tokens.</div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#parseNextToken()">parseNextToken</a>():<a href="../parser/Token.html">Token</a>
</div>
<div class="summaryTableDescription"></div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#skipToken()">skipToken</a>(token:<a href="Token.html">Token</a>):<a href="../parser/Token.html">Token</a>
</div>
<div class="summaryTableDescription"></div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
<tr class="">
<td class="summaryTablePaddingCol">&nbsp;</td><td class="summaryTableInheritanceCol">&nbsp;</td><td class="summaryTableSignatureCol">
<div class="summarySignature">
<a class="signatureLink" href="#tokenWasSkipped()">tokenWasSkipped</a>(token:<a href="Token.html">Token</a>):Boolean</div>
<div class="summaryTableDescription"></div>
</td><td class="summaryTableOwnerCol">Tokenizer</td>
</tr>
</table>
</div>
<script type="text/javascript" language="javascript">
<!--
showHideInherited();
--></script>
<div class="MainContent">
<a name="propertyDetail"></a>
<div class="detailSectionHeader">Property detail</div>
<a name="rawData"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">rawData</td><td class="detailHeaderType">property</td>
</tr>
</table>
<div class="detailBody">
<code>rawData:String</code>&nbsp;&nbsp;[read-write]<p>
   
   
   </p><span class="label">Implementation</span>
<br>
<code>&nbsp;&nbsp;&nbsp;&nbsp;public function get rawData():String</code>
<br>
<code>&nbsp;&nbsp;&nbsp;&nbsp;public function set rawData(value:String):void</code>
<br>
</div>
<a name="token"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">token</td><td class="detailHeaderType">property</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected var token:<a href="../parser/Token.html">Token</a></code><p>
   The token that was most recently created
   </p></div>
<a name="constructorDetail"></a>
<div class="detailSectionHeader">Constructor detail</div>
<a name="Tokenizer()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">Tokenizer</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">constructor</td>
</tr>
</table>
<div class="detailBody">
<code>public function Tokenizer()</code>
</div>
<a name="methodDetail"></a>
<div class="detailSectionHeader">Method detail</div>
<a name="addTokenPattern()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">addTokenPattern</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td>
</tr>
</table>
<div class="detailBody">
<code>public function addTokenPattern(pattern:<a href="TokenPattern.html">TokenPattern</a>):void</code><p>
   Adds the definition of a new TokenPattern, used for parsing the input string into tokens
   </p><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">pattern</span>:<a href="TokenPattern.html">TokenPattern</a></code></td>
</tr>
</table>
</div>
<a name="buildRegExpORList()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">buildRegExpORList</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected static function buildRegExpORList(str:String):String</code><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">str</span>:String</code></td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code>String</code></td>
</tr>
</table>
</div>
<a name="buildRegExpORList2()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">buildRegExpORList2</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected static function buildRegExpORList2(... rest):String</code><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code>... <span class="label">rest</span></code></td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code>String</code></td>
</tr>
</table>
</div>
<a name="createToken()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">createToken</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected function createToken(pattern:<a href="TokenPattern.html">TokenPattern</a>, value:String, sourceIndex:int, prev:<a href="Token.html">Token</a> = null):<a href="../parser/Token.html">Token</a></code><p>
   Factory method for Tokens. Override and extend in subclasses. Return null to indicate
   that the token is discarded, so the tokenizer will try to parse another one.
   </p><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">pattern</span>:<a href="TokenPattern.html">TokenPattern</a></code></td>
</tr>
<tr>
<td class="paramSpacer">&nbsp;</td>
</tr>
<tr>
<td width="20px"></td><td><code><span class="label">value</span>:String</code></td>
</tr>
<tr>
<td class="paramSpacer">&nbsp;</td>
</tr>
<tr>
<td width="20px"></td><td><code><span class="label">sourceIndex</span>:int</code></td>
</tr>
<tr>
<td class="paramSpacer">&nbsp;</td>
</tr>
<tr>
<td width="20px"></td><td><code><span class="label">prev</span>:<a href="Token.html">Token</a></code> (default = <code>null</code>)<code></code></td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/Token.html">Token</a></code></td>
</tr>
</table>
</div>
<a name="getState()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">getState</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public function getState():<a href="../parser/TokenizerState.html">TokenizerState</a></code><p>
   Captures the state of  the tokenizer so that it may be restored later.
   </p><p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/TokenizerState.html">TokenizerState</a></code></td>
</tr>
</table>
</div>
<a name="getTokenMetrics()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">getTokenMetrics</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public function getTokenMetrics(token:<a href="Token.html">Token</a>):<a href="../parser/TokenMetrics.html">TokenMetrics</a></code><p>
   This is not an optimized way of getting this information; it's really just for debugging
   and error messages
   </p><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">token</span>:<a href="Token.html">Token</a></code></td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/TokenMetrics.html">TokenMetrics</a></code></td>
</tr>
</table>
</div>
<a name="lookAhead()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">lookAhead</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public function lookAhead(offset:int = 1):<a href="../parser/Token.html">Token</a></code><p>
   Get a subsequent token, without affecting the tokenizer state
   </p><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">offset</span>:int</code> (default = <code>1</code>)<code></code> &mdash; int The number of tokens to look ahead
   </td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/Token.html">Token</a></code> &mdash; 
                        the token that is ahead from the current token by the specified amount
   
                        
                     </td>
</tr>
</table>
</div>
<a name="nextToken()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">nextToken</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public final function nextToken():<a href="../parser/Token.html">Token</a></code><p>
   </p><p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/Token.html">Token</a></code> &mdash; 
                        the next token in the stream, or throw an EOFError if there are no more valid tokens
   
                        
                     </td>
</tr>
</table>
</div>
<a name="parseNextToken()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">parseNextToken</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected function parseNextToken():<a href="../parser/Token.html">Token</a></code>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/Token.html">Token</a></code></td>
</tr>
</table>
</div>
<a name="reset()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">reset</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public function reset():void</code><p>
   Start getting tokens from the beginning
   </p></div>
<a name="restoreState()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">restoreState</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public function restoreState(state:<a href="TokenizerState.html">TokenizerState</a>):void</code><p>
   restores the tokenizer state to a memento(TM) created from getState()
   </p><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">state</span>:<a href="TokenizerState.html">TokenizerState</a></code></td>
</tr>
</table>
</div>
<a name="skipAhead()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">skipAhead</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>public function skipAhead(numTokens:int = 1):void</code><p>
   Skip the specified number of tokens.
   </p><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">numTokens</span>:int</code> (default = <code>1</code>)<code></code></td>
</tr>
</table>
</div>
<a name="skipToken()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">skipToken</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected function skipToken(token:<a href="Token.html">Token</a>):<a href="../parser/Token.html">Token</a></code><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">token</span>:<a href="Token.html">Token</a></code></td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code><a href="../parser/Token.html">Token</a></code></td>
</tr>
</table>
</div>
<a name="tokenWasSkipped()"></a>
<table cellspacing="0" cellpadding="0" class="detailHeader">
<tr>
<td class="detailHeaderName">tokenWasSkipped</td><td class="detailHeaderParens">()</td><td class="detailHeaderType">method</td><td class="detailHeaderRule">&nbsp;</td>
</tr>
</table>
<div class="detailBody">
<code>protected function tokenWasSkipped(token:<a href="Token.html">Token</a>):Boolean</code><span class="label">Parameters</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20px"></td><td><code><span class="label">token</span>:<a href="Token.html">Token</a></code></td>
</tr>
</table>
<p></p>
<span class="label">Returns</span>
<table border="0" cellspacing="0" cellpadding="0">
<tr>
<td width="20"></td><td><code>Boolean</code></td>
</tr>
</table>
</div>
<br>
<br>
<hr>
<br>
<p></p>
<center class="copyright"> 
</center>
</div>
</body>
</html>
<!-- -->
